Modeling and Representing Religious Language to Support Audio Transcription of Sermons

In a text-based discovery and analytical environment, high quality textual representation is needed to support discovery of and research on spoken content. The increased representation of human thoughts and ideas as digitally represented speech highlights the need for efficient generation of high quality text representations of spoken content. The most cost-effective method of producing textual representations is speech recognition systems. While much progress has been made in speaker-dependent (e.g., speaker trained) speech recognition systems, they produce poor quality results when applied in domain agnostic and speaker independent contexts (e.g., digitally recorded spoken content posted to the web). Results generated by domain-agnostic and speaker-independent language models are not usable for discovery or analysis. The poor quality results are due in part to the misalignment of domain-specific vocabularies and the domain-agnostic dictionaries used for acoustic pattern matching in speech recognition systems. The field of speech recognition is complex. Language models comprise only one of the four major components of speech recognition systems. Current speech recognition systems use language models which typically represent a non-domain specific vocabulary of 1,000 words. This is considered to be a large language space in speech recognition systems. This paper reports on exploratory research designed to test quality improvements that may be achieved by developing domain-focused phonemic vocabularies. The research relies on human knowledge engineering methods to model domain-specific languages. The research leverages the Atlas.ti application to extract and model religious language. The Logios application is used to convert the text vocabulary of 25,000 words to phonemic representation. The research focuses on digitally recorded spoken religious sermons as the test corpus. The value of the research is in identifying MODELING AND REPRESENTING RELIGIOUS LANGUAGE TO SUPPORT AUDIO TRANSCRIPTION what appears to be one common sense explanation for the poor differences in the language models of the religious domain and the generic language models that MODELING AND REPRESENTING RELIGIOUS LANGUAGE TO SUPPORT AUDIO TRANSCRIPTION Research Context Improving Access to and Transcription of Spoken Language Print culture has dominated western society for the past 500 years, playing a prime role in the effort to document human thought and ideas. Despite this important role, speech remains the preferred means of communicating thought and ideas Ong 1958) (Ong 1982). Recent developments in digital technology, specifically low cost digital recorders, smart phones and cameras, have made it possible for human thought and ideas to be effectively and efficiently documented not only in writing but in speech. For knowledge scientists, the expanded scope of digital representations presents significant opportunities. Spoken language is of particular interest to knowledge sciences because it is the primary form of knowledge transfer, mobilization and consumption and the means through which tacit knowledge becomes explicit. While our ability to capture the spoken word has increased, our ability to discover, access and analyze spoken content has not. Discovery, access, and analytical technologies are still text based. In a text-based information culture, we must have a high quality text version of the spoken content. This limits our ability to access and analyze thoughts and ideas that have been expressed orally. This is an important limitation to address in a world where spoken content may become the preferred means of communicating. Today there are four common methods (Table 1) that support access to spoken content, including: (1) direct human consumption through listening; (2) human transcription; (3) human tagging; and (4) machine transcription using speech recognition systems. Each method comes with tradeoffs in terms of quality and costs. Direct human listening produces the highest quality representation as there is no intermediary and no opportunity for misinterpretation or misrepresentation. This method, though, comes at a high costs. Direct costs include the time that an individual spends listening. The spoken content must be consumed one user at a time. MODELING AND REPRESENTING RELIGIOUS LANGUAGE TO SUPPORT AUDIO TRANSCRIPTION Any research conducted is by definition liable to subjective interpretation. There is no persistent textual representation that might be consumed by others. The second method, human transcription, offers high quality textual representation. The first copy costs are very high, though, due to the amount of human labor required and the cost of that labor, where the spoken content represents a specialized field or topic. Where the transcription is created for a single individual, it may not be accessible to others. Economies of scale may not be possible. The third method, human tagging, supports a primitive level of access by providing some access points for text-based indexing systems. However, the tags do not provide sufficiently reliable content to support objective analytical research. Tags are highly subjective, and are dependent upon some degree of listening. Research may be performed on the tags and their characteristics, but they do not serve as a reliable source of research on the spoken content. The fourth method, machine transcription through speech recognition systems, offers the best hope for providing high quality low cost textual representations of spoken content (Rabiner 1993). While we have decades of research devoted to speech recognition technologies that have resulted in some improvements in constrained contexts (e.g., single user speech trained dictionaries), the quality of the results is low. The challenges of speech recognition must be addressed if we expect to be able to conduct research on spoken context in the future. Table 1. Methods of Textual Representation of Audio Content Textual Representation Method Quality, Cost and Access Implications Direct End User Listening Highest quality representation, high user cost, no economies of scale, no persistent text representation Human Transcription Very high first-copy cost, high quality (assuming transcriber has knowledge of domain), copyrighted product, limited access due to pricing Tagging Moderate human cost, quality is poor and subjective, MODELING AND REPRESENTING RELIGIOUS LANGUAGE TO SUPPORT AUDIO TRANSCRIPTION Textual Representation Method Quality, Cost and Access Implications standalone tags, no persistent textual representation of content Machine-based Transcription Low cost, low quality, typically not widely used due to the low quality and need for significant quality control Research Context The essence of the challenge we face was well stated by John Pierce of Bell Labs (1969) when speech recognition systems were in their infancy. Pierce suggested that high quality speech would not be achieved until we had the capability to embed human intelligence and linguistic competence into the technologies. The challenges involved in producing high quality representation of audio content are as complex as human communication. This paper reports on exploratory research into one small facet of the speech recognition process – language modeling. Following Pierce’s 1969 advice, this research considers how embed a small component of human intelligence and linguistic competence – domain modeled vocabularies into speech recognition systems. For exploratory purposes we focus on the modeling of religious language to support the machine transcription of orally delivered sermons. In order to understand what we mean by language modeling, we offer a conceptual description of the context for speech recognition. We take Jelinek’s (1997) Source-Channel Model as our conceptual description. Jelinek identifies four components that are required to support speech recognition, including: (1) acoustic processing; (2) acoustic modeling; (3) language modeling; and (4) acoustic pattern matching. Acoustic processing addresses the transformation of the waveform into symbols. The raw audio signal needs to be converted into a sequence of frames at regular time intervals which may represent words or phrases. People listening to audio implicitly break the incoming wave forms into pieces that may represent semantic units – words (Allen 1994). The machine simply hears MODELING AND REPRESENTING RELIGIOUS LANGUAGE TO SUPPORT AUDIO TRANSCRIPTION acoustic symbols which may be music or speech or any other form of audio content. Acoustic processing is first step provides a foundation for assigning semantic meaning to audio content (Peller Proakis Hansen 1993) (Rabiner Schlafer 1978). The second component of Jelinek’s model addresses acoustic modeling. Acoustic modeling addresses the ability of the acoustic processor to create a semantic unit that is close to representing a word string. Acoustic modeling works at the phonemic level (Schultz Waibel 2001). A phoneme is an abstract unit represented in a phonetic system of language that corresponds to the sounds of speech. We will talk more about this below in a discussion of the phonemic representation of the language of sermons. Phonemic modeling based on the sounds of speech takes into account pronunciation, dialects and accents. The third component of the model focuses on language modeling. Language models consider how words are combined into larger semantic units (e.g. phrases, sentences, commands, etc.). There are two main types of language models in use today in speech recognition systems, including: (1) statistical language models (SLM); and (2) constrained grammars. Statistical language models help us to understand common use of language for the purpose of predicting language patterns for acoustical matching (Bahl et al 1993) (Bourland Hermansky Morgan 1996). A constrained grammar contains a precise definition of every phrase that can be recognized how a telephone number can be spoken, complex command and control systems, or high probability language patterns within a domain. The constrained grammar can be used when we have a good understanding of the language that is likely to be used in the context. The majority of the research into language modeling focuses on statistical language models (e.g. Hidden Markov Models) (Lippman 1990). This research focuses on the use of constrained grammars and considers how knowledge of the nature of a domain language might improve the quality of transcription. MODELING AND REPRESENTING RELIGIOUS LANGUAGE TO SUPPORT AUDIO TRANSCRIPTION Today’s speech recognition systems offer the opportunity to right size the language model to the content. Speech recognition systems will always match against the language base phonemic matching is at such a primitive level. We believe that what is lacking is the phonemic representation of domain-specific languages against which speech recognition systems can discover a good and relevant match. Acoustic pattern matching is the fourth component of Jelinek’s model. Acoustic pattern matching focuses on how the system goes about finding that good match – finding an accurate phonemic representation of the spoken word. Acoustic pattern matching leverages a phonemic translation of a word base. It processes the acoustically generated phonemes against the phonemes in the base dictionary. To the extent that we are not optimizing the relevance of the language model there will not be a good foundation for matching. It is generally the case that the language model dictionary is not domain specific. Research Goal and Hypothesis Most high quality results of speech recognition applications are based on speaker dependent training. These applications require a speaker to train the system before an acceptable level of quality can be expected. Speaker dependent speech recognition generates a phonemic language model that can effectively constrain the range for acoustic pattern matching. This approach is inconvenient, less robust, more wasteful, and not feasible for some applications. Speaker dependent training of speech recognition systems is not feasible for spoken content posted to the web for the simple reason that the speaker is not available to create a phonemic dictionary. As a next-best option for bounding the acoustic matching challenge, we propose to create domain specific phonemic dictionaries. MODELING AND REPRESENTING RELIGIOUS LANGUAGE TO SUPPORT AUDIO TRANSCRIPTION We believe that two challenges arise when looking for a good match for acoustically processed and modeled speech, specifically: (1) the large unbounded language space, and (2) the phonemic representation of the unbounded space. We suggest that this approach may be semantically inefficient (i.e., broader than necessary) and ineffective (i.e., it fails to leverage domain expert knowledge). Current approaches to modeling large vocabularies (e.g. 1000 word dictionaries) attempt to constrain the language space using domain agnostic text analytics, phrase and sentence extraction. We suggest that this is semantically inefficient. To increase the efficiency we suggest that domain-specific language models will provide a more relevant base for acoustic pattern matching. We also suggest that what is commonly considered a large vocabulary for speech recognition purposes does not align with the size of domain-specific language. We look to human knowledge engineering and semantic analysis methods to design domain-specific language models. We suggest that human engineered domain-specific vocabularies will produce higher quality results by creating a more relevant and targeted language against which to conduct acoustic patterns matching. Improvements to the quality of transcription make our fourth approach to representation cost effective. Research Data and Methodology To explore these ideas, we selected a collection of digital recordings of spoken sermons. We selected religious sermons because there is a need and an interest to provide textual representations for research and analytical purposes. Religious sermons also represent a domain-specific vocabulary which could be semantically profiled and validated. This research methodology was exploratory, complex and integrative. It was designed to address four research questions, each providing a foundation for the next. The four questions were: MODELING AND REPRESENTING RELIGIOUS LANGUAGE TO SUPPORT AUDIO TRANSCRIPTION • Question 1: Is there a domain-specific language for religious information? • Question 2: How might we integrate a religious language model into speech recognition systems? • Question 3: Does the integration of a religious language model into speech recognition systems contribute to an increase in quality? • Question 4: Is this model applicable to other domains? Research Question 1 – Domain Language of Religion Our research goal for Question 1 was to understand how well the standard language models used in speech recognition systems aligned with the language of religious sermons. To answer this question, we adopted a three step methodology to investigate this research question which involved: (1) creating and semantically processing a corpus of sermons representing the language of religion; (2) analyzing the linguistic patterns and syntactical properties of the language of sermons; and (3) analyzing the implications of these language models for quality speech recognition of religious sermons. A corpus of text materials was created to analyze the grammatical structure the language of religion and religious sermons (Step 1). The corpus was comprised of more than five hundred sermon texts (Bedford 2011) (Bedford 2012), several versions of the Bible, digital copies of religious education materials including teaching templates and syllabi, and content drawn from church websites including Audio Sermons Online, Good Shepherd Lutheran Church Sermon Recordings, Hope Church Sermon Recordings, and Sermon Audio. The corpus was first processed using optical character recognition (OCR). The Atlas.ti application was used to extract individual words from the full corpus. The result was a vocabulary of 24,691 unique words. We note the significant variation from what is commonly characterized as a large vocabulary (e.g. MODELING AND REPRESENTING RELIGIOUS LANGUAGE TO SUPPORT AUDIO TRANSCRIPTION 1,000 words) in the speech recognition field. From this vocabulary we created a grammatical profile of the language of religion. The profile produced a language model that varied slightly from the language models that we found are used as dictionaries in speech recognition systems. We would like to acknowledge an additional challenge in finding a typical language profile against which to validate. In fact any aspect of adult language appears to be context or domain specific. There may not be a good characterization beyond primary or secondary school language use. Table 2. Grammatical Characterization of the Language of Religion Natural Language Tag Part of Speech % Representation in the Corpus ADJ Adjective 16.26% ADV Adverb 14.19% CONJ Conjunction 2.77% PREP Preposition 6.92% PRON Pronoun 6.57% VERBS Verbs 28.03% NOUNS Nouns 25.26% 100% From this initial exploration, it would appear that there are variations in sentence patterns (Step 3). The language of religion is characterized by a low use of nouns, a slightly low use of verbs, a very high use of adverbs and adjectives, and a very high use of pronouns. Given the variations, we suggest that the use of a domain-specific dictionary as a replacement for the generic dictionaries in most speech recognition systems may produce higher quality acoustic pattern matching. Research Question 2: How might we integrate a religious language model into speech recognition systems? MODELING AND REPRESENTING RELIGIOUS LANGUAGE TO SUPPORT AUDIO TRANSCRIPTION The exploration of this research question required a five-step methodology, including: (1) creating a clean representation of the religious vocabulary, including reducing OCR errors; (2) creating a phonemic representation of the vocabulary; (3) assessing the coverage of the generic dictionaries used for language modeling; (4) defining an effective phonemic language model for the domain; and (5) constructing the revised language model and testing it in a speech recognition system against a test set of audio recordings of sermons. Creating a clean copy of the religious vocabulary involved walking through all 24,691 words to identify errors. We encountered and corrected several OCR generated errors. Errors were generated where the OCR application could not recognize the fonts used in historical sermons and versions of the Bible. Extraneous punctuation also had to be cleaned from the vocabulary. Graphics included in some texts also produced errors. As was noted earlier, the language model of a speech recognition system requires a phonemic representation of the vocabulary. Speech recognition systems use one of two phonemic alphabets: the International Phonetic Association’s IPAbet (2005) or DARPA’s ARPAbet. IPAbet is an alphabetic system of phonetics produced by the International Phonetic Association which focuses on the Latin alphabet. It provides a standardized representation of the sounds of spoken or oral language. ARPAbet is a phonetic transcription code which was developed by the Advanced Research Projects Agency as part of their Speech Understanding Project from 1971 to 1976 (Shoup 1980). IPAbet was chosen for this research because that is the alphabet used by the Carnegie Mellon University Sphinx application (Carnegie Mellon University Speech Group 2008) (Lee Hon Reddy 1990). The translation of the religious vocabulary was supported by Carnegie Mellon University’s Logios application (Carnegie Mellon University 2008) (Weide 1998). Logios uses hand-tuned linguistic rules creates by expert linguists. Sample translations are MODELING AND REPRESENTING RELIGIOUS LANGUAGE TO SUPPORT AUDIO TRANSCRIPTION provided in Tables 2a, 2b and 2c. Table 3a. Sample Phonemic Representation of the Language of Religion Words from Sermon Extractions Phonemic Pronunciation AARON AA R AH N ABANDON AH B AE N D AH N ABANDONED AH B AE N D AH N D ABANDONMENT AH B AE N D AH N M AH N T ABEL EY B AH L ABERNATHY AE B ER N AE TH IY ABETS AH B EH T S ABETTING AH B EH T IH NG ABEYANCE AH B EY AH N S ABHOR AE B HH AO R ABHORRED AH B HH AO R D ABIDE AH B AY D ABIDES AH B AY D Z ABIDING AH B AY D IH NG ABILITY AH B IH L AH T IY ABJECT AE B JH EH K T Table 3b. Sample Phonemic Representation of the Language of Religion Words from Sermon Extractions Phonemic Pronunciation ENDLESS EH N D L AH S ENTERS EH N T ER Z ETERNAL IH T ER N AH L FINISH F IH N IH SH FOLLOW F AA L OW FOR F AO R FORGIVE F ER G IH V FOXES F AA K S AH Z GO G OW GOD G AA D GOOD G UH D GRANT G R AE N T HA HH AA HAIL HH EY L HAT HH AE T HE HH IY HEART HH AA R T MODELING AND REPRESENTING RELIGIOUS LANGUAGE TO SUPPORT AUDIO TRANSCRIPTION Words from Sermon Extractions Phonemic Pronunciation HEIRS EH R Z HERE HH IY R Table 3c. Sample Phonemic Representation of the Language of Religion Words from Sermon Extractions Phonemic Pronunciation TO T UW TON T AH N TONIGHT T AH N AY T TOO T UW TORY T AO R IY TRULY T R UW L IY TRY T R AY TWO T UW USE Y UW S VANITY V AE N AH T IY VENGEANCE V EH N JH AH N S VICTORY V IH K T ER IY VILE V AY L WAS W AA Z WE W IY WE’VE W IY V WELL W EH L The initial analysis (Step 3) indicates the Logios application had to construct a phonemic representation of about 9%-10% of the vocabulary. A large percentage of the words in the vocabulary were found in the main Logios Phonemic Dictionary. This suggests that we can leverage the existing IPAbet (International Phonetic Association 2005) or ARPAbet repositories for translating domain vocabularies to phonemic representation. For the religious vocabulary, we must expect to create new entries for about 10% of the terms. In conducting this research, we learned that the Sphinx speech recognition system uses the Wall Street Journal’s standard vocabulary. Our research suggests that using that full vocabulary creates an unnecessarily large and inefficient vocabulary for matching religious language MODELING AND REPRESENTING RELIGIOUS LANGUAGE TO SUPPORT AUDIO TRANSCRIPTION materials (Step 4). We believe that an intelligently constrained and bounded language model that is representative of the domain will produce quality improvements. The next step in our research is to replace the existing dictionary with a domain specific vocabulary. Given what we understand about the grammatical profile of religious language, we intend to supplement the vocabulary adverbs and adjectives. We expect that our revised constrained grammar model comprise about 30,000 words. The revised vocabulary will be converted to IPAbet phonemic representation. Research Question 3: Does the integration of a religious language model into speech recognition systems contribute to an increase in quality? Preliminary tests produced small improvements in quality when a domain vocabulary was used. However, it seemed that the generic dictionary based on the Wall Street Journal’s standard vocabulary might be confounding the matching. We reconsidered the dictionary architecture. We suggest that it will be important to remove and replace the embedded dictionary with the domain specific vocabulary. Because the architecture of speech recognition systems is complex, we hope to work with system developers to accomplish this next step. We also learned that open source applications provide opportunities for understanding the mechanics of speech recognition and for varying parameters for testing. However, we also observed that these applications may not be as robust as commercial systems. We learned that we need to test our work in both open source and commercial systems. Research Question 4: Is this model applicable to other domains? Might this research be expanded to other domain-specific language models? We believe that if we can produce quality improvement with this approach, it will be applicable to other domains. We believe that the more peculiar a language model is to a domain, the greater promise MODELING AND REPRESENTING RELIGIOUS LANGUAGE TO SUPPORT AUDIO TRANSCRIPTION this approach offers. Research Findings and Observations The value of this research would appear to be the importance of leveraging human knowledge engineering and semantic analysis methods to produce domain-specific language to support language modeling in speech recognition. While speech recognition systems consider large vocabularies spaces to include 1,000 words, we found a large space for a religious vocabulary in upwards of 25,000 words. We also found that there are grammatical variations in the use of language across domains. To not leverage that knowledge may result in an inefficient language model design. We believe that a domain-agnostic approach to supporting language models in speech recognition is inefficient from an acoustic matching perspective. ‘ Most speech recognition systems allow the swapping in and out of dictionaries to support language modeling. The follow-on research will explore the most effective dictionary architecture. For example, should multiple domain-specific dictionaries be architected into speech recognition systems as distinct dictionaries? Should domain specific dictionaries be accessible upon demand in a speech recognition system? We believe that a two part process may help to identify the domain-specific language model to invoke to support speech recognition processing. If we know that we’re working with an audio recording of a sermon, we should be able to invoke a religious language model to improve quality of transcription. The follow-on research will also explore whether human generated tags of spoken content can help to automatically detect the domain language model to use. Observations and Future Research We intend to apply the research methodology to four new domains, including transportation, education, agriculture and finance. The test for quality improvements will be applied to audio MODELING AND REPRESENTING RELIGIOUS LANGUAGE TO SUPPORT AUDIO TRANSCRIPTION files across all five domains, including the corpus of religious sermons. We expect to conduct the tests working with commercially available speech recognition products and under the guidance of speech recognition application developers. MODELING AND REPRESENTING RELIGIOUS LANGUAGE TO SUPPORT AUDIO TRANSCRIPTION Biographical Statement Denise Bedford is currently the Goodyear Professor of Knowledge Management at the College of Communication and Information, Kent State University. She teaches courses in knowledge management, communities of practice, economics of information, semantic analysis, enterprise architecture, business intelligence, organizational network analysis, and information environments. Her educational background includes a B.A. in History, in Russian Language/Literature, and in German Language/Literature from the University of Michigan; an M.A.in Russian History also from University of Michigan; an M.S. in Librarianship from Western Michigan University, and a Ph.D. in Information Science from University of California, Berkeley. MODELING AND REPRESENTING RELIGIOUS LANGUAGE TO SUPPORT AUDIO TRANSCRIPTION